Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix #5430: Fix flakes in NetworkLoggingInterceptorTest #5436

Merged
merged 1 commit into from
Jun 21, 2024
Merged

Fix #5430: Fix flakes in NetworkLoggingInterceptorTest #5436

merged 1 commit into from
Jun 21, 2024

Conversation

BenHenning
Copy link
Member

@BenHenning BenHenning commented Jun 21, 2024 •
edited
Loading

Explanation

Fixes #5430

Fix flaky tests in NetworkLoggingInterceptorTest and (theoretically) in ConsoleLoggerTest.

See #5430 for investigation methodology and findings. Ultimately, the problem was that tests were trying to observed SharedFlow results after changes to that flow may have been emitted (which means there's no guarantees by the flow that the value is present upon trying to consume it, hence the flake in the job consuming the flow not completing).

Note that ConsoleLoggerTest's fix is only theoretical since, having only one test, it doesn't seem to ever produce the correct situations that would cause the flake.

Originally, NetworkLoggingInterceptorTest would flake with almost a 20% failure rate. I've verified that the test passes without failures over 1000x runs:

> bazel test --runs_per_test=1000 //data:src/test/java/org/oppia/android/data/backends/gae/NetworkLoggingInterceptorTest

Target //data:src/test/java/org/oppia/android/data/backends/gae/NetworkLoggingInterceptorTest up-to-date:
  bazel-bin/data/src/test/java/org/oppia/android/data/backends/gae/NetworkLoggingInterceptorTest.jar
  bazel-bin/data/src/test/java/org/oppia/android/data/backends/gae/NetworkLoggingInterceptorTest
INFO: Elapsed time: 833.435s, Critical Path: 103.87s
INFO: 1001 processes: 1 internal, 1000 linux-sandbox.
INFO: Build completed successfully, 1001 total actions
//data:src/test/java/org/oppia/android/data/backends/gae/NetworkLoggingInterceptorTest PASSED in 103.3s
  Stats over 1000 runs: max = 103.3s, min = 38.1s, avg = 77.7s, dev = 12.4s

Executed 1 out of 1 test: 1 test passes.

Similarly, I've also checked the change in ConsoleLoggerTest just to ensure nothing's changed:

> bazel test --runs_per_test=1000 //utility/src/test/java/org/oppia/android/util/logging:ConsoleLoggerTest

Target //utility/src/test/java/org/oppia/android/util/logging:ConsoleLoggerTest up-to-date:
  bazel-bin/utility/src/test/java/org/oppia/android/util/logging/ConsoleLoggerTest.jar
  bazel-bin/utility/src/test/java/org/oppia/android/util/logging/ConsoleLoggerTest
INFO: Elapsed time: 767.614s, Critical Path: 105.49s
INFO: 1001 processes: 1 internal, 1000 linux-sandbox.
INFO: Build completed successfully, 1001 total actions
//utility/src/test/java/org/oppia/android/util/logging:ConsoleLoggerTest PASSED in 104.6s
  Stats over 1000 runs: max = 104.6s, min = 31.3s, avg = 71.6s, dev = 8.8s

Executed 1 out of 1 test: 1 test passes.

Essential Checklist

  • The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
  • Any changes to scripts/assets files have their rationale included in the PR explanation.
  • The PR follows the style guide.
  • The PR does not contain any unnecessary code changes from Android Studio (reference).
  • The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
  • The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

This PR only affects the behavior of two test suites; no end user UX or UIs are affected by these changes.

@BenHenning
Fix flakes in NetworkLoggingInterceptorTest.
18d255d 
This also fixes theoretical flakes in ConsoleLoggerTest.
@BenHenning BenHenning marked this pull request as ready for review June 21, 2024 02:07
@BenHenning BenHenning requested a review from a team as a code owner June 21, 2024 02:07
BenHenning
BenHenning commented Jun 21, 2024
View reviewed changes
Copy link
Member Author

@BenHenning BenHenning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-reviewed the fixes.

@BenHenning
Copy link
Member Author

@adhiamboperes PTAL for codeowners.

I'm also enabling auto-merge since I hope this isn't a change that requires a lot of iteration (though if you have thoughts, please do let me know).

@BenHenning BenHenning enabled auto-merge (squash) June 21, 2024 02:09
@github-actions GitHub Actions
Copy link

APK & AAB differences analysis

Note that this is a summarized snapshot. See the CI artifacts for detailed differences.

Dev

Expand to see flavor specifics

Universal APK

APK file size: 16 MiB (old), 16 MiB (new), 0 bytes (No change)

APK download size (estimated): 14 MiB (old), 14 MiB (new), 1 bytes (Removed)

Method count: 221791 (old), 221791 (new), 0 (No change)

Features: 2 (old), 2 (new), 0 (No change)

Permissions: 6 (old), 6 (new), 0 (No change)

Resources: 6550 (old), 6550 (new), 0 (No change)

  • Anim: 49 (old), 49 (new), 0 (No change)
  • Animator: 20 (old), 20 (new), 0 (No change)
  • Array: 15 (old), 15 (new), 0 (No change)
  • Attr: 915 (old), 915 (new), 0 (No change)
  • Bool: 10 (old), 10 (new), 0 (No change)
  • Color: 911 (old), 911 (new), 0 (No change)
  • Dimen: 994 (old), 994 (new), 0 (No change)
  • Drawable: 373 (old), 373 (new), 0 (No change)
  • Id: 1206 (old), 1206 (new), 0 (No change)
  • Integer: 37 (old), 37 (new), 0 (No change)
  • Interpolator: 11 (old), 11 (new), 0 (No change)
  • Layout: 368 (old), 368 (new), 0 (No change)
  • Menu: 1 (old), 1 (new), 0 (No change)
  • Mipmap: 1 (old), 1 (new), 0 (No change)
  • Plurals: 10 (old), 10 (new), 0 (No change)
  • Raw: 2 (old), 2 (new), 0 (No change)
  • String: 805 (old), 805 (new), 0 (No change)
  • Style: 816 (old), 816 (new), 0 (No change)
  • Xml: 6 (old), 6 (new), 0 (No change)

Lesson assets: 105 (old), 105 (new), 0 (No change)

AAB differences

Expand to see AAB specifics

Supported configurations:

  • hdpi (same)
  • ldpi (same)
  • mdpi (same)
  • tvdpi (same)
  • xhdpi (same)
  • xxhdpi (same)
  • xxxhdpi (same)

Base APK

APK file size: 15 MiB (old), 15 MiB (new), 0 bytes (No change)
APK download size (estimated): 14 MiB (old), 14 MiB (new), 19 bytes (Removed)

Configuration hdpi

APK file size: 59 KiB (old), 59 KiB (new), 0 bytes (No change)
APK download size (estimated): 23 KiB (old), 23 KiB (new), 0 bytes (No change)

Configuration ldpi

APK file size: 56 KiB (old), 56 KiB (new), 0 bytes (No change)
APK download size (estimated): 18 KiB (old), 18 KiB (new), 0 bytes (No change)

Configuration mdpi

APK file size: 53 KiB (old), 53 KiB (new), 0 bytes (No change)
APK download size (estimated): 18 KiB (old), 18 KiB (new), 0 bytes (No change)

Configuration tvdpi

APK file size: 102 KiB (old), 102 KiB (new), 0 bytes (No change)
APK download size (estimated): 38 KiB (old), 38 KiB (new), 0 bytes (No change)

Configuration xhdpi

APK file size: 67 KiB (old), 67 KiB (new), 0 bytes (No change)
APK download size (estimated): 28 KiB (old), 28 KiB (new), 0 bytes (No change)

Configuration xxhdpi

APK file size: 76 KiB (old), 76 KiB (new), 0 bytes (No change)
APK download size (estimated): 38 KiB (old), 38 KiB (new), 0 bytes (No change)

Configuration xxxhdpi

APK file size: 79 KiB (old), 79 KiB (new), 0 bytes (No change)
APK download size (estimated): 39 KiB (old), 39 KiB (new), 0 bytes (No change)

Alpha

Expand to see flavor specifics

Universal APK

APK file size: 10 MiB (old), 10 MiB (new), 0 bytes (No change)

APK download size (estimated): 9150 KiB (old), 9150 KiB (new), 9 bytes (Added)

Method count: 99436 (old), 99436 (new), 0 (No change)

Features: 2 (old), 2 (new), 0 (No change)

Permissions: 6 (old), 6 (new), 0 (No change)

Resources: 5504 (old), 5504 (new), 0 (No change)

  • Anim: 39 (old), 39 (new), 0 (No change)
  • Animator: 18 (old), 18 (new), 0 (No change)
  • Array: 14 (old), 14 (new), 0 (No change)
  • Attr: 879 (old), 879 (new), 0 (No change)
  • Bool: 8 (old), 8 (new), 0 (No change)
  • Color: 767 (old), 767 (new), 0 (No change)
  • Dimen: 722 (old), 722 (new), 0 (No change)
  • Drawable: 333 (old), 333 (new), 0 (No change)
  • Id: 1148 (old), 1148 (new), 0 (No change)
  • Integer: 32 (old), 32 (new), 0 (No change)
  • Interpolator: 11 (old), 11 (new), 0 (No change)
  • Layout: 327 (old), 327 (new), 0 (No change)
  • Menu: 1 (old), 1 (new), 0 (No change)
  • Mipmap: 1 (old), 1 (new), 0 (No change)
  • Plurals: 10 (old), 10 (new), 0 (No change)
  • String: 736 (old), 736 (new), 0 (No change)
  • Style: 457 (old), 457 (new), 0 (No change)
  • Xml: 1 (old), 1 (new), 0 (No change)

Lesson assets: 105 (old), 105 (new), 0 (No change)

AAB differences

Expand to see AAB specifics

Supported configurations:

  • hdpi (same)
  • ldpi (same)
  • mdpi (same)
  • tvdpi (same)
  • xhdpi (same)
  • xxhdpi (same)
  • xxxhdpi (same)

Base APK

APK file size: 9 MiB (old), 9 MiB (new), 0 bytes (No change)
APK download size (estimated): 9030 KiB (old), 9030 KiB (new), 2 bytes (Added)

Configuration hdpi

APK file size: 52 KiB (old), 52 KiB (new), 0 bytes (No change)
APK download size (estimated): 22 KiB (old), 22 KiB (new), 0 bytes (No change)

Configuration ldpi

APK file size: 52 KiB (old), 52 KiB (new), 0 bytes (No change)
APK download size (estimated): 17 KiB (old), 17 KiB (new), 0 bytes (No change)

Configuration mdpi

APK file size: 46 KiB (old), 46 KiB (new), 0 bytes (No change)
APK download size (estimated): 17 KiB (old), 17 KiB (new), 0 bytes (No change)

Configuration tvdpi

APK file size: 90 KiB (old), 90 KiB (new), 0 bytes (No change)
APK download size (estimated): 37 KiB (old), 37 KiB (new), 0 bytes (No change)

Configuration xhdpi

APK file size: 60 KiB (old), 60 KiB (new), 0 bytes (No change)
APK download size (estimated): 27 KiB (old), 27 KiB (new), 0 bytes (No change)

Configuration xxhdpi

APK file size: 69 KiB (old), 69 KiB (new), 0 bytes (No change)
APK download size (estimated): 37 KiB (old), 37 KiB (new), 0 bytes (No change)

Configuration xxxhdpi

APK file size: 71 KiB (old), 71 KiB (new), 0 bytes (No change)
APK download size (estimated): 38 KiB (old), 38 KiB (new), 0 bytes (No change)

Beta

Expand to see flavor specifics

Universal APK

APK file size: 9 MiB (old), 9 MiB (new), 4 bytes (Removed)

APK download size (estimated): 9135 KiB (old), 9135 KiB (new), 12 bytes (Added)

Method count: 99436 (old), 99436 (new), 0 (No change)

Features: 2 (old), 2 (new), 0 (No change)

Permissions: 6 (old), 6 (new), 0 (No change)

Resources: 5504 (old), 5504 (new), 0 (No change)

  • Anim: 39 (old), 39 (new), 0 (No change)
  • Animator: 18 (old), 18 (new), 0 (No change)
  • Array: 14 (old), 14 (new), 0 (No change)
  • Attr: 879 (old), 879 (new), 0 (No change)
  • Bool: 8 (old), 8 (new), 0 (No change)
  • Color: 767 (old), 767 (new), 0 (No change)
  • Dimen: 722 (old), 722 (new), 0 (No change)
  • Drawable: 333 (old), 333 (new), 0 (No change)
  • Id: 1148 (old), 1148 (new), 0 (No change)
  • Integer: 32 (old), 32 (new), 0 (No change)
  • Interpolator: 11 (old), 11 (new), 0 (No change)
  • Layout: 327 (old), 327 (new), 0 (No change)
  • Menu: 1 (old), 1 (new), 0 (No change)
  • Mipmap: 1 (old), 1 (new), 0 (No change)
  • Plurals: 10 (old), 10 (new), 0 (No change)
  • String: 736 (old), 736 (new), 0 (No change)
  • Style: 457 (old), 457 (new), 0 (No change)
  • Xml: 1 (old), 1 (new), 0 (No change)

Lesson assets: 105 (old), 105 (new), 0 (No change)

AAB differences

Expand to see AAB specifics

Supported configurations:

  • hdpi (same)
  • ldpi (same)
  • mdpi (same)
  • tvdpi (same)
  • xhdpi (same)
  • xxhdpi (same)
  • xxxhdpi (same)

Base APK

APK file size: 9 MiB (old), 9 MiB (new), 8 bytes (Removed)
APK download size (estimated): 9019 KiB (old), 9019 KiB (new), 5 bytes (Removed)

Configuration hdpi

APK file size: 52 KiB (old), 52 KiB (new), 0 bytes (No change)
APK download size (estimated): 22 KiB (old), 22 KiB (new), 0 bytes (No change)

Configuration ldpi

APK file size: 52 KiB (old), 52 KiB (new), 0 bytes (No change)
APK download size (estimated): 17 KiB (old), 17 KiB (new), 0 bytes (No change)

Configuration mdpi

APK file size: 46 KiB (old), 46 KiB (new), 0 bytes (No change)
APK download size (estimated): 17 KiB (old), 17 KiB (new), 0 bytes (No change)

Configuration tvdpi

APK file size: 90 KiB (old), 90 KiB (new), 0 bytes (No change)
APK download size (estimated): 37 KiB (old), 37 KiB (new), 0 bytes (No change)

Configuration xhdpi

APK file size: 60 KiB (old), 60 KiB (new), 0 bytes (No change)
APK download size (estimated): 27 KiB (old), 27 KiB (new), 0 bytes (No change)

Configuration xxhdpi

APK file size: 69 KiB (old), 69 KiB (new), 0 bytes (No change)
APK download size (estimated): 37 KiB (old), 37 KiB (new), 0 bytes (No change)

Configuration xxxhdpi

APK file size: 71 KiB (old), 71 KiB (new), 0 bytes (No change)
APK download size (estimated): 38 KiB (old), 38 KiB (new), 0 bytes (No change)

Ga

Expand to see flavor specifics

Universal APK

APK file size: 9 MiB (old), 9 MiB (new), 4 bytes (Removed)

APK download size (estimated): 9136 KiB (old), 9136 KiB (new), 16 bytes (Added)

Method count: 99436 (old), 99436 (new), 0 (No change)

Features: 2 (old), 2 (new), 0 (No change)

Permissions: 6 (old), 6 (new), 0 (No change)

Resources: 5504 (old), 5504 (new), 0 (No change)

  • Anim: 39 (old), 39 (new), 0 (No change)
  • Animator: 18 (old), 18 (new), 0 (No change)
  • Array: 14 (old), 14 (new), 0 (No change)
  • Attr: 879 (old), 879 (new), 0 (No change)
  • Bool: 8 (old), 8 (new), 0 (No change)
  • Color: 767 (old), 767 (new), 0 (No change)
  • Dimen: 722 (old), 722 (new), 0 (No change)
  • Drawable: 333 (old), 333 (new), 0 (No change)
  • Id: 1148 (old), 1148 (new), 0 (No change)
  • Integer: 32 (old), 32 (new), 0 (No change)
  • Interpolator: 11 (old), 11 (new), 0 (No change)
  • Layout: 327 (old), 327 (new), 0 (No change)
  • Menu: 1 (old), 1 (new), 0 (No change)
  • Mipmap: 1 (old), 1 (new), 0 (No change)
  • Plurals: 10 (old), 10 (new), 0 (No change)
  • String: 736 (old), 736 (new), 0 (No change)
  • Style: 457 (old), 457 (new), 0 (No change)
  • Xml: 1 (old), 1 (new), 0 (No change)

Lesson assets: 105 (old), 105 (new), 0 (No change)

AAB differences

Expand to see AAB specifics

Supported configurations:

  • hdpi (same)
  • ldpi (same)
  • mdpi (same)
  • tvdpi (same)
  • xhdpi (same)
  • xxhdpi (same)
  • xxxhdpi (same)

Base APK

APK file size: 9 MiB (old), 9 MiB (new), 0 bytes (No change)
APK download size (estimated): 9020 KiB (old), 9020 KiB (new), 17 bytes (Added)

Configuration hdpi

APK file size: 52 KiB (old), 52 KiB (new), 0 bytes (No change)
APK download size (estimated): 22 KiB (old), 22 KiB (new), 0 bytes (No change)

Configuration ldpi

APK file size: 52 KiB (old), 52 KiB (new), 0 bytes (No change)
APK download size (estimated): 17 KiB (old), 17 KiB (new), 0 bytes (No change)

Configuration mdpi

APK file size: 46 KiB (old), 46 KiB (new), 0 bytes (No change)
APK download size (estimated): 17 KiB (old), 17 KiB (new), 0 bytes (No change)

Configuration tvdpi

APK file size: 90 KiB (old), 90 KiB (new), 0 bytes (No change)
APK download size (estimated): 37 KiB (old), 37 KiB (new), 0 bytes (No change)

Configuration xhdpi

APK file size: 60 KiB (old), 60 KiB (new), 0 bytes (No change)
APK download size (estimated): 27 KiB (old), 27 KiB (new), 0 bytes (No change)

Configuration xxhdpi

APK file size: 69 KiB (old), 69 KiB (new), 0 bytes (No change)
APK download size (estimated): 37 KiB (old), 37 KiB (new), 0 bytes (No change)

Configuration xxxhdpi

APK file size: 71 KiB (old), 71 KiB (new), 0 bytes (No change)
APK download size (estimated): 38 KiB (old), 38 KiB (new), 0 bytes (No change)

adhiamboperes
adhiamboperes approved these changes Jun 21, 2024
View reviewed changes
Copy link
Collaborator

@adhiamboperes adhiamboperes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, thanks for investigating!

@BenHenning BenHenning merged commit 14976d9 into develop Jun 21, 2024
26 checks passed
@BenHenning BenHenning deleted the fix-network-logging-interceptor-test-flakes branch June 21, 2024 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adhiamboperes adhiamboperes adhiamboperes approved these changes

Assignees

@adhiamboperes adhiamboperes

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG]: Flaky Test: testLoggingInterceptor_makeNetworkCall_succeeds
2 participants
@BenHenning @adhiamboperes